The present invention relates to stereoscopic three-dimensional (hereinafter 3D) video.
A conventional 3D viewing system will now be described with additional reference to FIGS. 1-7.
FIG. 1 illustrates the processing of stereoscopic imagery by the brain.
As shown in the figure, a person 100 has a head 102. Head 102 includes a left eye 114, a right eye 116 separated by a horizontal displacement 120 and a brain 104. Brain 104 includes an image fuser 110.
Brain 104 is operable to store a left image 106 received from left eye 114 and a right image 108 received from right eye 116. Left image 106 represents an image of image 118 as viewed by left eye 114. Right image 108 represents an image of image 118 as viewed by right eye 116. Image fuser 110 is operable to combine left image 106 and right image 108 to create a 3D image 112.
In operation, person 100 uses left eye 114 to receive left image 106 and right eye 116 to receive right image 108. Left image 106 and right image 108 are received and processed to generate and store image 118 in brain 104. Left eye 114 is horizontal displacement 120 away from right eye 116. Because of horizontal displacement 120, left image 106 of image 118 captured by left eye 114 is horizontally different from right image 108 of image 118 captured by right eye 116. In brain 104, left image 106 is horizontally displaced from right image 108. In combination, left image 106 and right image 108 form a stereoscopic image pair. Image fuser 110 is operable to combine left image 106 and right image 108 into a composite, 3D image 112. Using left image 106 and right image 108, brain 104 is better operable to perceive imagery with depth and perspective.
The operation of a conventional camera assembly will now be described with additional reference to FIG. 2.
FIG. 2 illustrates the capture and storage of stereoscopic imagery by a conventional 3D camera assembly.
As shown in the figure, a conventional 3D camera assembly 200 includes a left camera 202, a right camera 204, a memory portion 208 and a memory portion 222.
Memory portion 208 is operable to store imagery captured by left camera 202. Memory portion 222 is operable to store imagery captured by right camera 204.
Left camera 202 is separated from right camera 204 by horizontal displacement 224.
Memory portion 208 includes a left image 206, a left image 208, a left image 210 and a left image 212, whereas memory portion 222 includes a right image 214, a right image 216, a right image 216 and a right image 220. Images 206, 208, 210 and 212 of memory portion 208 are arranged in a pair with an image 214, 216, 218 and 220, respectively, of memory portion 222 to form an image pair 223, an image pair 224, an image pair 226 and an image pair 228, respectively.
Conventional 3D camera assembly 200 is arranged to approximate brain 104's capacity to capture and store stereoscopic image pairs. Similar to left eye 114 and right eye 116 receiving and storing horizontally displaced imagery, left camera 202 and right camera 204 also receive and store horizontally displaced imagery because left camera 202 is arranged horizontal displacement 224 from right camera 204. Therefore, as left camera 202 and right camera 204 capture and store left images 212, 210, 208 and 206 and right images 220, 218, 216 and 214, respectively, of image 118, stereoscopic image pairs 228, 226, 224 and 222 are created due to horizontal displacement 224 between left camera 202 and right camera 204. In one example, stereoscopic image pair 228 includes left image 212 and right image 220. Left image 212 and right image 220 of stereoscopic image pair 228 are horizontally shifted from one another as a function of horizontal displacement 224 between left camera 202 and right camera 204.
The operation of a person receiving and processing stereoscopic data by way of a conventional 3D image controller will now be described with additional reference to FIG. 3.
FIG. 3 illustrates person 100 receiving stereoscopic image data from a conventional stereoscopic image source 300 by way of a conventional 3D image controller 302.
As shown in the figure, stereoscopic image source 300 has left and right image collections 208 and 222, respectively, each containing the respective left images 206, 208, 210 and 212 and right images 214, 216, 218 and 220 of image pairs 223, 224, 226 and 228, 3D image controller 302 includes a left image controller 304 and a right image controller 306.
3D image controller 302 is arranged to receive images from stereoscopic image source 300. In particular, left image controller 304 is arranged to receive left image collection 208 from stereoscopic image source 300 for selective transmission to left eye 114. Whereas, right image controller 306 is arranged to receive right image collection 222 from stereoscopic image source 300 for selective transmission to right eye 116.
Person 100 is able to receive images from 3D image controller 302. In particular, left eye 114 is arranged to sequentially receive images 212, 210, 208 and 206 of left image collection 208 from stereoscopic image source 300 as transmitted by left image controller 304. In addition, right eye 116 is arranged to sequentially receive images 220, 218, 216 and 214 of right image collection 222 from stereoscopic image source 300 as transmitted by right image controller 306.
Brain 104 is operable to receive and store left image 106 from left eye 114 and right image 108 from right eye 116 image fuser 110 is operable to merge left image 106 and right image 108 into a composite, 3D image 112.
The stereoscopic image pairs 228, 226, 224 and 223 of stereoscopic image source 300 can be created using conventional 3D camera assembly according to the description of FIG. 2. 3D image controller 302 is used to sequentially transmit the left images of left image collection 208 to left eye 114 and to sequentially transmit the right images of right image collection 222 to right eye 116 thereby fixing left image 106 to correspond, to left image collection 208 and fixing right image 108 to correspond to right image collection 222.
Brain 104 may experience discomfort when image fuser 110 operates on right image 108 and left image 106 is arranged using an un-natural horizontal displacement 224.
3D image controller 302 may include left image controller 304 and right image controller 306. In one example, left image controller 304 includes a projection device arranged to sequentially project the images of left image collection 208 using a blue light and includes a first portion of a set of eye glasses wherein the first portion is a blue lens arranged to be projectionally interposed between left eye 114 and left image collection 208. Continuing the example, right image controller 306 includes a projection device arranged to sequentially project the images of right image collection 222 using a red light and includes a second portion, of a set of eyeglasses wherein the second portion is a red lens arranged to be projectionally interposed between right eye 116 and right image collection 222. In this example, the blue projected left images 206, 208, 210 and 212 are received by left eye 114 due to the filtering effect of left image controller 304 and the red projected right images 214, 216, 218 and 220 are received by the right eye 116 due to the filtering effect of right image controller 306.
The components of stereoscopic imagery will now be described and exemplified with additional references to FIGS. 4A and 4B.
FIGS. 4A-B illustrate a left image 400 and a right image 402 respectively.
As shown in FIG. 4A, left image 400 includes a pattern 406. As shown in FIG. 4B, right image 402 includes a pattern 408.
Parallax will now be described and exemplified with additional reference to FIG. 4C.
FIG. 4C illustrates a parallax 410 of left image 400 and right image 402 as an example of stereoscopic imagery.
As shown in the figure, a stereoscopic image 404 includes pattern 406 superimposed onto pattern 408. Parallax 410 is the horizontal offset between patterns 406 and 408.
FIG. 4C illustrates parallax 410, which is the horizontal displacement between two images of the same object taken from different horizontal positions. Horizontal displacement 120 between left eye 114 and right eye 116 and horizontal displacement 224 between left camera 202 and right camera 204 are operable to create parallax 410 between captured images such as pattern 406 and pattern 408. For example, left eye 114 may capture left image 400 having pattern 406 and right eye 116 may capture right image 402 having pattern 408 wherein pattern 406 and 408 are actually two different views of the same pattern horizontally offset by parallax 410 from one another. Image fuser 110 is operable to convert stereoscopic image 404 into a single 3D image 112.
Positive and negative parallax will now be described and exemplified with additional reference to FIGS. 5 and 6.
FIG. 5 illustrates an example of a negative parallax.
FIG. 5 includes left eye 114, right eye 116, an image 506, a right image 520, a left image 522, a viewing plane 516, a plane of convergence 514, a positive fusion offset 524, a focus distance 518, a display screen 500, a perceived image point 508, a right image point 510 and a left image point 512.
As shown in the figure, right image 520 and left image 522 are projected onto display screen 500. Left eye 114 and right eye 116 focus on display screen 500. Left eye 114 and right eye 116 are focus distance 518 away from viewing plane 516 and move to converge for receiving right image 520 and left image 522. Image 506 is therefore perceived to be in front of display screen 500. More specifically, image 506 is the perceived image created by image fuser 110 as described with reference to FIG. 1 appearing in front of display screen 500 on plane of convergence 514 at positive fusion offset 524 from display screen 500.
3D effects are perceptions created by image fuser 110 when operating on left and right imagery arranged in unnatural parallax. Unnatural parallax includes when a left image is to the right of a right image. As shown in FIG. 5, left image 522 is projected to the right of right image 520 on display screen 500 creating an unnatural parallax. 3D image controller 302 (not shown) is operable to insure that left eye 114 captures left image 522 and that right eye 116 captures right image 520.
Because left eye 114 is to the left of right eye 116, left eye 114 does not capture a left image such as left image 522 horizontally displaced to the right of a right image such as right image 520 in the natural world. Image fuser 110 is designed to fuse left images. The left images are horizontally displaced to the left with right images horizontally displaced to the right.
When left eye 114 captures left image 522 physically projected to the right of right image 520 on display screen 500 and right eye 116 captures right image 520 physically projected to the left of left image 522 on display screen 500, image fuser 110 creates image 506 having a negative parallax. Image 506 is a perception created by image fuser 110. Left image 522 and right image 520 are the physical images. Because of the unnatural relative positions of right image 520 and left image 522, image 506 is perceived by brain 104 to be in front of display screen 500 by positive fusion offset 524 on plane of convergence 514. Plane of convergence 514 and positive fusion offset 524 are examples of convergence data.
Generally speaking, a plane of convergence may be considered a plane where the left and right images are shifted toward one another so as to converge into a single image. With respect to FIG. 5, left image 522 viewed by left eye 114 and right image 520 viewed by right eye 116 verge so as to converge into a single image. As such, with respect to the discussion of FIG. 5 (and the following discussion of FIG. 6) a plane of convergence may be considered a plane of vengeance.
FIG. 6 illustrates an example of positive parallax.
FIG. 6 includes left eye 114, right eye 116, an image 600, a right image 608, a left image 610, a viewing plane 516, a plane of convergence 606, a display screen 500, a perceived image point 604, a right image point 602 and a left image point 612.
As shown in the figure, right image 608 and left image 610 are protected onto display screen 500. Left eye 114 and right eye 116 focus on display screen 500 which is focus distance 518 away from viewing plane 516 and move to converge for receiving right image 608 and left image 610. Image 600 is the perceived image created by image fuser 110 (not shown) appearing behind display screen 500 on plane of convergence 606 at a negative fusion offset 620 from display screen 500. Plane of convergence 606 and negative fusion offset 620 are examples of convergence data.
As shown in FIG. 6, left image 610 is projected to the left of right image 608. This arrangement of horizontal displacement between images is known as positive parallax. When left eye 114 receives left image 610 and right eye 116 receives right image 608 arranged in positive parallax, image fuser 110 combines the images such that brain 104 perceives the combined image to be beyond the screen display 500 by a distance of negative fusion offset 620 as shown b perceived image point 604 in the figure.
FIG. 5 and FIG. 6 illustrate image manipulation not found in nature. In nature, images appear on a convergence plane positioned at focus distance 518. However, image pairs projected onto display screen 500 are perceived to be positive fusion offset 524 or negative fusion offset 620 away from display screen 500. Focus distance 518 is the distance between viewing plane 516 and display screen 500. In many cases, the physical location of projected image pairs of any parallax are focus distance 518 away from viewing plane 516 because the projected images are physically projected on the screen. However, because of positive or negative parallax, image fuser 110 perceives the image to be either closer or farther away than focus distance 518 on a plane of convergence different from display screen 500. In nature the plane of convergence is focus distance 518 away from viewing plane 516. Viewing discomfort including tiredness, headache, eyestrain, dizziness, and the like can be caused when brain fuser 110 needs to process left image 106 and right image 108 to create 3D image 112 appearing on a convergence plane not positioned at focus distance 518 away from viewing plane 516. Moreover, viewing discomfort may be further worsened by large, abrupt, rapid, or frequent changes to the position of the plane of convergence.
A conventional 3D image stream and corresponding fusion offset time plot will now be described and exemplified with additional reference to FIG. 7.
FIG. 7 illustrates as multi-scene 3D image stream 700 and a corresponding plot of the fusion offset for each instance of stereoscopic data.
As shown in FIG. 7, 3D multi-scene 3D image stream 700 includes a scene 702, a scene 716 and a scene 730.
As shown in the figure, scene 702 contains a frame 704, a frame 708 and a frame 712 each having an image pair which together form an instance of stereoscopic data. The stereoscopic data of each frame 704, 708 and 712 includes a fusion offset 706, 710 and 714, respectively. Scene 716 and scene 730 are composed similarly to scene 702.
Also as shown in FIG. 7, a plot 748 has a y-axis 744 and an x-axis 746. Plot 748 includes points 706, 710, 714, 724, 726, 728, 738, 740 and 742.
In operation, multi-scene 3D image stream 700 is sequentially projected onto display screen 500. Typical of most movies, multi-scene 3D image stream 700 is composed of many scenes with a sampling noted as scene 702, scene 716 and scene 730 wherein each scene is composed or several frames, each frame being composed of an image pan having a parallax measured by fusion offset resulting in a perceived image position on a convergence plane. For example, frame 704 of scene 702 has a negative parallax measured by fusion offset 706 which means that the image created by image fuser 110 will be perceived to be on a convergence plane in front of display screen 500.
Also shown in FIG. 7, an abrupt change to the value of two sequential fusion offsets 714 and 724 may occur between the last frame of a first scene and the first frame of a second scene.
Likewise, an abrupt change to the value of fusion offset occurs between frame 722 of scene 716 and frame 732 of scene 730 as shown by plotted fusion offset values 728 and 738.
If care is not taken during the manufacture of a 3D movie, frequent abrupt changes to the convergence plane may occur between frames of different scenes resulting in viewing discomfort. Recall that 3D image capture may include 2 cameras positioned horizontally apart as shown in FIG. 2 for filming the same scene or image each from a slightly different perspective to approximate the slightly different images captured b each eye of a pair of human eyes.
In a first example, if left camera 202 and right camera 204 are not positioned correctly, the left and right images may be significantly more horizontally displaced than would be if captured by a pair of human eyes. In this case, the brain and the muscles that control the eyes may need to do additional work and may have to work faster in order to focus on and combine the two images into the single image noted as 3D image 112, which brain 104 perceives. The additional work can be the cause of stress, fatigue, pain, discomfort, and lack of enjoyment.
In a second example, left camera 202 and right camera 204 may not exceed normal boundaries of horizontal displacement 224. Even so, carelessness or lack of planning during postproduction may result in a 3D image stream as shown in FIG. 7 having frequent and abrupt changes to fusion offset.
More particularly, consider a movie, film, or video such as multi-scene 3D image stream 700 as shown in FIG. 7 composed of many scenes such as scene 702, scene 716 and scene 730 with each scene potentially captured independently from the others at different times, using different equipment, and created by different teams. Without meticulous and expensive record keeping on the part of the production staff, changes to fusion offset such as a max 745 or a max 746 are likely to occur between the last frame of a scene and the first frame of the subsequent scene.
Rapid, successive changes to fusion offset results also in rapid, successive changes to the plane of convergence as shown in FIG. 9 a plane of convergence 924 and a plane of convergence 928.
Recall in the natural world plane of convergence 514 is focus distance 518 from viewing plane 516 meaning that the human eye perceives images at their physical distance meaning, that right image 108 and left image 106 have zero parallax. When viewing stereoscopic imagery created with positive parallax as shown in FIG. 6 or negative parallax as shown in FIG. 5, left eye 114, right eye 116, and brain 104 operate differently than normal. The different operation may be one cause oft) image stream view discomfort.
Stereoscopic imagery having a non-zero parallax requires brain 104 to perceive combined 3D image 112 at location, plane of convergence 514, 606 that is different from the actual physical location of right image 520, 608 and left image 610, 522, which is focus distance 518, away from viewing plane 516. When brain 104, left eye 114, and right eye 116 are forced to focus on an object at a location different from its perceived location, viewing discomfort may be created.
Moreover, 3D image viewing characterized by sequential stereoscopic image manipulation resulting in rapid and frequent changes of positive or negative parallax as shown in FIG. 5 and FIG. 6 causes left eye 114 and right eye 116 to continuously move at angles consistent with the plane of convergence whilst focusing at a distance corresponding to the actual physical locations of the sequential images at distances different from the planes of convergence.
Fatigue, eyestrain, and headaches are reported results of 3D image viewing. This discomfort can be attributed to the effects on the eyes and brain processing manufactured unnatural stereoscopic images having non-zero parallax.
Fatigue, eyestrain, and headaches are compounded as the brain processes a continuous stream of sequentially manufactured unnatural stereoscopic images characterized by rapid and abrupt parallax changes between image frames resulting in rapid and abrupt fusion offset and plane of convergence changes.
What is needed is a system and method that reduces the change in parallax, fusion offset, and convergence planes between successive stereoscopic image frames whilst preserving the novel and enjoyable experience of watching 3D imagery to reduce or eliminate fatigue, eyestrain, headaches, and other films of discomfort.